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Abstract 

Previous  research  in  this  area  has  developed,  tested,  and  implemented  a  system  maturity  measure,  i.e. 
System  Readiness  Level  (SRL);  supporting  optimization  models;  and  an  enhanced  SRL  hierarchy  for  multi¬ 
function,  multi-capability  (MFMC)  systems.  The  later  developments  are  predicated  on  what  has  become 
the  accustomed  challenge  for  managers  and  engineers  to  properly  assess  systems'  development  and 
acquisition  to  ensure  the  achievement  of  critical  capabilities  and  functions  while  deciding  amongst 
multiple  technologies  with  similar  functionalities  but  different  maturity  levels  and  limited  resources. 
Building  upon  these  developments,  this  research  asked:  How  can  we  efficiently  and  effectively  allocate 
available  resources  to  ensure  the  maturity  achievement  of  critical  functions  and  capabilities  in  an  MFMC 
system  when  facing  competing  technology  alternatives?  To  address  this  question,  this  work  developed  a 
multi-objective  optimization  model  and  solution  approach  that  can  be  used  to  evaluate  systems' 
development  maturity,  track  progress,  identify  component  criticality,  and  form  corresponding  strategies 
for  understanding  trade-offs  in  technology  and  integration  options. 
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1.  Introduction 

Under  the  direction  of  the  Principal  Investigators  (Pis)  and  with  support  from  the  Naval  Postgraduate 
School  (NPS)  and  government/industry  partnerships,  the  Systems  Development  and  Maturity  Laboratory 
(SysDML)  at  Stevens  Institute  of  Technology  has  developed  the  following: 

a)  A  methodology  for  determining  a  system's  development/acquisition  maturity  (i.e.,  System 
Readiness  Level  [SRL];  Sauser,  Ramirez-Marquez,  Henry,  &  DiMarzio,  2008;  Sauser  &  Ramirez- 
Marquez,  2009b;  NPS  BAA  07-002); 

b)  Two  optimization  models  for  allocating  resources  so  as  to  optimize  cost,  schedule,  and  maturity 
performance  (i.e.,  SRLmax  and  SCODmin;  Ramirez-Marquez  &  Sauser,  2009;  Sauser  and  Ramirez- 
Marquez,  2009b;  Magnaye,  Sauser,  &  Ramirez-Marquez,  2010;  NPS  BAA  07-002); 

c)  A  methodology  that  combines  items  (a)  and  (b)  into  an  approach  called  Systems  Earned 
Readiness  Management  (SERM;  Magnaye,  Sauser,  &  Ramirez-Marquez,  2009;  Sauser  &  Ramirez- 
Marquez,  2009;  Magnaye,  2010;  NPS  BAA  08-004); 

d)  Models  to  determine  which  components1  are  sufficient,  critical,  or  important  to  achieve  a  level 
of  system  maturity  for  the  intended  functionalities  and  capabilities  of  a  system  (Tan  et  al.,  2010; 
NPS  BAA  09-002);  and 

e)  An  enhanced  SRL  hierarchy  that  enables  multiple  architectural  views  in  maturity  assessment  at 
the  capability,  function,  and  system  levels  and  incorporates  an  Analysis  of  Alternatives  (AoA;  or 
trade-off)  for  technology  and  integration  component  adoption  which  impacts  the  system 
maturity,  functionality,  and  capability  (Tan,  Sauser,  &  Ramirez-Marquez,  2011a,  2011b;  NPS  BAA 
10-002). 

It  is  important  to  note  that  throughout  these  research  efforts,  the  SysDML  has  maintained  an 
evolutionary  development  through  which  we  have  worked  closely  with  industry  and  government  to 
refine  and  implement  our  research  in  order  to  ensure  its  relevance  and  rigor  (see  Exhibit  1  for  a 
summary).  For  example,  current  efforts  funded  by  the  Acquisition  Research  Program  (ARP)  are 
addressing  some  recurring  issues  that  were  revealed  through  conversations  with  our  industry  and 
government  research  partners.  One  issue  of  great  interest  to  us  is  the  following: 

...  with  increasing  development  and  acquisition  of  systems  that  are  built  upon  open  and 
flexible  platform  designs  that  accommodate  multiple  functions  and  capabilities  as  well 
as  the  ability  for  adopting  future  mission  packages  (e.g.  modularity,  system  of  systems), 
the  decisions  to  trade-off  among  multiple  alternatives  that  enable  necessary  functions 
and  capabilities  will  be  unavoidable. 

While  the  recent  efforts  in  AoA  are  necessary  and  relevant  to  the  understanding  of  systems  maturity 
and  alternatives  comparison  and  prioritization,  its  application  is  limited  in  an  acquisition  situation  where 


1  Components  are  defined  as  the  technologies  and/or  integrations  of  the  system  of  interest. 
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in  the  procurement  of  a  complex  system  there  are  many  technology  alternatives  with  the  same 
functionality  but  different  maturity  states  or  different  cost  request  for  further  maturation.  Therefore, 
this  research  built  upon  the  previous  AoA  research  to  propose  the  employment  of  multi-objective  (MO) 
optimization  models  for  the  development  of  systems  that  can  perform  multiple  functions  and  multiple 
capabilities  (MFMC). 

Contrary  to  the  single-objective  optimization  models  developed  in  previous  research,  this  effort 
addressed  the  MO  problem  and  provides  potential  solutions  to  assist  managers  with  flexibility  in  the 
planning  of  system  acquisition  and  further  system  maturation.  Thus,  this  research  addresses  a 
fundamental  question: 


How  can  we  efficiently  and  effectively  allocate  available  resources  to  ensure  the 
maturity  achievement  of  critical  functions  and  capabilities  in  an  MFMC  system  when 
facing  competing  technology  alternatives? 


Analysis  of 
Alternatives  MFMC 


System 
Capability 
sficing 


NPS  BAA-11-002 
•  efficiently  and  effectively  allocate 
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•  determine  the 
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and  its  status  within 
the  acquisition  life 
cycle. 


Exhibit  1.  Evolutionary  Development 


2.  Purpose  and  Focus  of  Research 

Over  the  past  six  years,  the  SRL  scale  and  supporting  methods  have  undergone  continuous  development 
and  have  been  accepted  as  a  valid  metric  to  measure  the  readiness  of  a  system  throughout  its 
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development  lifecycle  within  various  organizations  (e.g.,  U.S.  Navy-PMS420,  U.S.  Army  Armament 
Research,  Development  and  Engineering  Center  [ARDEC],  Lockheed  Martin,  National  Security  Agency, 
and  Northrop  Grumman).  However,  to  date,  this  scale  has  focused  on  the  management  of  systems 
intended  to  perform  a  single  function.  Today,  even  the  most  basic  weapons  such  as  assault  rifles  have 
become  multi-functional.  As  such,  during  the  development  process,  these  systems  may  be  called  upon 
to  deliver  some  of  their  capabilities  even  as  others  are  still  several  phases  behind.  Often,  this  requires  an 
AoA  among  technology  choices  and  architectures  to  take  advantage  of  those  that  are  already  mature, 
though  not  originally  intended  for  use  in  the  development  of  the  desired  function.  This,  in  turn,  requires 
a  thorough  understanding  of  the  technical  aspects  of  the  components  but,  more  significantly,  the 
relative  importance  of  each  choice  on  the  readiness  of  the  system  vis-a-vis  its  desired  capability. 

In  practice,  a  system  evolves  with  time  from  a  single  capability  of  a  specific  function  to  a  more 
complicated  one  that  affords  multiple  functions  and  ensures  the  operational  performance  of  a  function 
by  having  several  backup  capabilities.  Moreover,  in  order  to  ensure  the  success  of  the  development  or 
acquisition  of  a  system,  even  for  a  specific  function,  it  is  common  to  have  one  or  several  backup 
capabilities.  In  addition,  in  the  evolution  of  systems  development,  the  advancement  of  technology 
options  is  progressing  faster  than  the  systems  themselves.  In  this  process,  the  following  questions  may 
arise:  Will  a  new,  more  functional  system  or  technology  supersede  the  old?  Has  the  system  or 
technology  become  inadequate  due  to  changes  in  other  systems  or  technologies?  Is  it  more  effective  to 
invest  in  the  development  of  new  technology  or  system?  Has  the  system  or  technology  lifetime  been 
shortened  by  recent  development?  What  is  a  robust  methodology  that  can  effectively  and  efficiently 
analyze,  compare,  and  trade-off  technology  alternatives? 

With  an  ever-increasing  complexity  of  the  systems  that  are  being  developed  and  realized,  multiple 
functions  and  capabilities  are  common  to  the  development  of  most  systems,  which  raises  the  need  to 
balance  development  objectives  when  facing  multiple  component  alternatives.  As  a  result,  managers 
require  metrics  that  enable  the  assessment  of  MFMC  system  developmental  maturity  to  manage  the 
potential  risks  posed  by  the  previous  questions  (Volkert,  2009). 

For  example,  Forbes,  Volkert,  Michaud,  Gentile,  and  Sondi  (2009)  have  described  a  system  with  six 
capabilities  that  are  realized  by  six  threads  of  components.  The  architecture  of  this  system  is 
represented  in  Exhibit  2.  As  described  in  their  paper,  this  system  had  undergone  a  system  maturity 
assessment  with  summary  charts  also  depicted  in  Exhibit  2.  The  initial  system  architecture  represented 
in  Exhibit  2(a)  resulted  in  an  assessment  with  an  overall  SRL  of  0.60  (see  MP  SRL  in  upper  right  box);  the 
identification  of  an  insufficiently  mature  technology  and  supporting  integrations  (circled);  and  analysis 
of  the  technology-integration  maturity  of  all  the  system  components  (horizontal  line  at  bottom  of 
diagram).  An  alternative  systems  solution  based  on  a  trade-off  (see  Exhibit  2(b))  considered  replacing  a 
single  technology  (i.e.,  MVCS)  with  another  two  technologies  (i.e.,  DLS  OB;  DLS  RMMV).  This  alternative 
does  not  significantly  improve  the  overall  SRL  value  (increase  of  only  0.04),  but  it  does  improve  the 
technology-integration  maturity  of  all  the  lagging  system  components  (horizontal  line  at  bottom  of 
diagram).  While  this  analysis  may  seem  sufficient  in  increasing  systems  maturity  toward  an  effective 
acquisition  decision,  it  does  not  consider  many  of  the  IMs  related  to  an  AoA  and  their  influence  on  the 
system's  current  or  future  maturity.  Thus,  a  decision  made  purely  on  an  increase  in  maturity  may  be 
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insufficient.  This  described  research  intends  to  address  this  concern  by  enhancing  the  SRL  approach  to 
take  into  account  the  multiple  system  functions  and  capabilities  which  allows  for  the  opportunity  to 


better  understand  an  AoA  in  technologies  and  integrations  to  more  effectively  manage  system  maturity 
and  acquisition. 
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3.  Research  Approach 

In  the  acquisition  of  systems,  it  is  a  core  objective  for  a  system  to  be  deployed  that  satisfies 
stakeholders'  needs  while  staying  within  acceptable  budgetary  and  schedule  constraints.  However, 
achieving  this  objective  has  never  been  easy  because  of  inherent  uncertainty,  which  has  been 
compounded  as  the  complexity  of  defense  systems  has  increased.  This  complexity  challenges  our  ability 
to  properly  manage  system  development  and  acquisition  in  face  of  the  lack  of  competent  techniques 
and  tools.  Therefore,  a  tension  exists  between  the  need  for  systems  development  and  our  actual 
capability  to  fulfill  this  need.  Although  impossible  to  eliminate,  the  tension  has  attracted  significant 
research  in  the  pursuit  of  tools  and  models  that  can  mitigate  it  to  some  degree. 

Mathematical  optimization  is  an  approach  used  to  maximize  or  minimize  a  system  figure  of  merit  (FOM; 
e.g.,  system  reliability,  acquisition  cost,  system  maturity)  in  search  of  the  optimal  solution  within  a 
feasible  resolution  space  (e.g.,  not  to  exceed  the  dedicated  resources  or  by  satisfying  other 
requirements).  For  optimization  to  be  effective,  it  is  vital  to  define  or  choose  the  appropriate  FOM.  The 
metric  of  Technology  Readiness  Level  (TRL)  is  a  FOM  that  has  been  used  across  many  U.S.  government 
agencies  serving  as  a  common  language  for  communicating  the  readiness  of  a  technology  before 
inserting  and  integrating  it  to  a  system  (Department  of  Defense  [DoD],  2009).  Since  its  debut  in  the 
National  Aeronautics  and  Space  Administration  (NASA),  TRL  has  gained  popularity  of  application,  and 
ample  efforts  have  been  devoted  to  better  articulating  its  use,  tailoring  it  for  the  development  of  other 
types  of  technologies,  or  following  its  logic  to  create  similar  metrics  for  the  use  in  other  domains  (Tan, 
Ramirez-Marquez,  &  Sauser,  2011). 

From  a  system  architecture  perspective,  any  system  is  composed  of  a  group  of  individual  technologies 
and  the  linkages  among  them,  so  two  types  of  knowledge  would  be  necessary  in  order  to  understand 
system  maturity:  component  knowledge  and  integration  knowledge  (Henderson  &  Clark,  1990). 
Therefore,  while  TRL  was  introduced  to  quantify  the  readiness  of  technologies  representing  component 
knowledge,  another  nine-level  metric  known  as  Integration  Readiness  Level  (IRL)  was  subsequently 
proposed  to  provide  integration  knowledge  that  measures  the  readiness  of  the  linkages  among 
technologies  (Gove,  2007;  Sauser,  Forbes,  Long,  &  McGrory,  2009;  Sauser,  Gove,  Forbes,  &  Ramirez- 
Marquez,  2010).  These  metrics  are  important  since  they  make  it  possible  for  various  stakeholders  to 
discuss  system  development-related  issues  on  a  common  language,  characterized  as  development 
maturity  that  is  quantified  and  expressed  as  readiness  levels.  With  the  two  standalone  metrics  created, 
the  introduction  of  a  comprehensive  metric  that  can  combine  these  two  metrics  and  put  them  in  the 
context  of  an  overall  system  development  would  be  beneficial.  In  the  response  to  filling  this  gap,  the 
metric  of  System  Readiness  Level  (SRL)  was  proposed  by  the  SysDML  as  a  numerical  product  of  the 
technology  TRLs  and  integration  IRLs  to  provide  system-level  maturity  assessment  (Sauser,  Ramirez- 
Marquez,  Magnaye,  &  Tan,  2008;  Ramirez-Marquez  &  Sauser,  2009). 

While  SRL  continues  to  be  verified  and  validated  with  systems  across  private  companies  and 
government  agencies,  there  are  endeavors  being  devoted  to  including  this  metric  in  optimization 
models  for  systems  development  planning  (e.g.,  SRLmax,  SCODmin,  ESM_SRLmax)  (Ramirez-Marquez  & 
Sauser,  2009;  Magnaye  et  al.,  2010;  Sauser,  Magnaye,  Tan,  Ramirez-Marquez,  &  Sauser,  2010).  In 
general,  these  models  provide  system  developers  and  managers  support  in  terms  of  enhancing  their 
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capabilities  on  decision-making  regarding  resource  allocation  during  the  systems'  development  and 
acquisition  lifecycle. 


These  models  have  advanced  the  use  of  SRL  to  a  more  dynamic  application;  however,  there  are  two 
aspects  that  have  not  been  taken  into  account  in  these  models,  which  limit  their  application.  First,  they 
limit  the  relevance  with  the  development  of  complex  systems  that  are  designed  and  deployed  to 
provide  multiple  functions  and  capabilities.  As  addressed  by  Tan,  Sauser,  and  Ramirez-Marquez  (2011b), 
often  systems  are  designed  and  developed  to  provide  multiple  functions  and  capabilities  with  a  single 
solution.  In  order  to  track  and  to  manage  the  development  of  these  MFMC  systems,  it  would  be 
insufficient  not  to  take  into  consideration  the  maturity  assessment  on  the  capability  and  function  levels. 
Especially  in  a  situation  where  some  of  the  capabilities  are  critical  for  development  because  of  the 
urgent  demand  on  the  market  or  in  an  on-going  war  theater,  the  need  to  measure  and  advance  the 
maturity  of  the  very  functions  and  capabilities  becomes  imperative,  without  which  the  development 
could  become  problematic.  To  solve  this  problem,  the  original  SRL  definition  was  enhanced  by  Tan, 
Sauser,  and  Ramirez-Marquez  (2011b)  to  include  two  layers  of  maturity  assessment  (i.e.,  the  function 
and  capability  layers)  and  enable  more  accurate  system  maturity  assessment.  See  Exhibit  3  as  an 
example  of  an  SRL  assessment  on  an  MFMC  system  architecture.  Compared  to  the  original  SRL 
formulation,  this  enhanced  SRL  provides  a  more  holistic  picture  and  better  insights  into  system  maturity 
assessment  because  the  system  maturity  analysis  is  more  systemic.  The  focus  is  not  on  a  myopic  view  of 
the  system  maturity,  but  on  multiple  architectural  perspectives  of  the  system.  However,  the 
aforementioned  optimization  models  were  based  on  the  original  SRL  and  therefore  fail  their  relevance 
to  accommodate  the  development  of  MFMC  systems. 


How  to  Use  this  Template 


1-input  the  date;  2-move  the  selector  in  the  System 
Development  Lifecycle  chart  to  choose  an  expected 
SRL  for  this  date;  3-specify  the  percentage  for  RED 
and  YELLOW  thresholds;  4-check  the  CAPABILITY  of 
interest;  5-for  scenario  analysis,  change  the  TRL/IRL 
in  the  graph. 


Date  MM/DD/YYYY 

Expected  SRL  O.&f 
Red  Bar 

Yellow  Bar  5%  ' 


n  SRL 

Function  1 


Capability  1.2 
Capability  1.3 
Capability  1.4 
Function  2 

Capability  2.1 


0. 

0.70 

0.70 

0.62 

0.70 

0.70 

0.70 


SRL 
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0.2 
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DoD  Life  Cycle 

0.5 


0.8  0.9  1.0 


I  I 


Materiel 

Technology 

Engineering  and 

Production 

Operations 

Solution 

Analysis 

Development 

Manufacturing 

Development 

& 

... 

&  Support 

o 


Pre- Systems  Acquisition 


Systems  Acquisition 


Sustainment 


Exhibit  3.  Multi-Function,  Multi-Capability  Systems  Maturity  Assessment 


Another  limitation  with  these  optimization  models  is  that  they  were  developed  from  a  single  FOM 
perspective;  that  is,  they  are  pursuing  the  optimization  of  a  single  objective  subject  to  a  set  of 
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constraints.  While  it  is  possible  to  repeatedly  run  a  model  in  order  to  get  the  optimal  solutions  under 
different  scenarios,  it  is  impossible  to  observe  the  interplay  between  two  or  more  conflicting  system 
attributes,  for  example,  how  the  maximum  SRL  changes  with  an  increase  in  development  resources.  In 
an  MFMC  system  development,  the  maturation  of  different  functions  or  capabilities  will  compete  for 
resources,  and  it  is  not  rare  where  several  critical  functions  and  capabilities  share  a  high  priority.  An 
additional  challenge  encountered  in  the  acquisition  of  systems  is  that  while  there  is  a  necessity  to 
dedicate  resources  for  system  development,  having  a  number  of  options  and  the  respective  required 
resources  to  advance  them  can  allow  for  trade-offs  between  the  systems  acquisition  budget  and 
potential  solutions.  Therefore,  it  is  beneficial  to  provide  multiple  solutions  from  which  acquisitionists 
can  have  the  flexibility  to  tailor  a  maturation  strategy  so  as  to  ensure  stakeholder  satisfaction. 
Therefore,  in  the  development  of  MFMC  systems,  the  more  common  desire  is  to  optimize  a  number  of 
objectives  concurrently,  where  MO  optimization  models  can  play  an  important  role. 

To  address  such  research  gaps,  this  research  employed  MO  optimization  techniques  to  accommodate 
the  development  of  MFMC  systems.  This  research  created  a  new  model  based  on  the  enhanced  SRL 
hierarchy  with  the  objective  to  maximize  capability,  function,  and  system  maturity  and  minimize  the 
consumption  of  development  resources. 

Model  MO_  SRL 

Max{{SRL_F(TRL,  IRL)},  {SRL_C(TRL,  IRL)},  SRL(TRL,  IRL)} 

Min  {RrfTRL,  IRL),  ...,  Rh(TRL,  IRL),...,  RH(TRL,  IRL)} 
s.t. 

TRLelTRL^TRL 9} 

IRLy  e  { lRLj ,  IRLy  + 1, . . . ,  9 } 


The  matrices  TRL  and  IRL  in  the  model  contain  the  decision  variables.  Each  of  these  variables  is  an 
integer,  bounded  by  [TRL,,  9]  or  [IRLy,  9],  respectively.  That  is,  any  TRL  or  IRL  cannot  be  less  than  its 
current  level  or  more  than  perfect  status,  which  is  Level  9  in  this  case.  This  model  includes  the 
optimization  of  two  types  of  objectives:  (i)  maximizing  the  SRLs  which  are  of  interest  to  managers  and 
(ii)  minimizing  resource  consumption.  The  first  type  of  optimization  as  listed  in  the  maximization  part  is 
that  each  SRL  (function,  capability,  or  system  SRL)  is  a  function  of  the  technology  TRLs  and  integration 
IRLs  of  the  system.  More  specifically,  the  function  and  capability  SRLs  are  determined  by  the  TRLs  and 
IRLs  of  the  relevant  technologies  and  integration  links,  which  constitute  the  function  or  capability  in 
question.  The  second  type  of  optimization  is  to  minimize  the  consumption  of  resources  under 
consideration  (for  example,  dollars,  labor  hours,  machinery,  etc.). 

In  order  to  completely  characterize  the  decision  variables  in  the  Model  MO_SRL,  the  following 
transformations  have  been  introduced: 

,  fl  If  TRL,  =t  ,  fl  lfIRL..=t 

y\=\  andx.=  1  forf=l,...9 

[0  otherwise  [0  otherwise 


9 


Notice  that  based  on  these  binary  variables,  each  of  the  possible  TRLs  and  IRLs  in  the  system  can  be 

9  9 

obtained  as:  TRLi  =  2>'  and  IRLii  =  ^tx'j  .  Therefore,  as  defined  in  the  SRL  hierarchy,  ITRLs, 


t= 1  t=i 

capability,  function,  and  system  SRLs  can  be  transformed  into  expressions  that  are  determined  by  these 
binary  variables.  For  example,  an  ITRL  of  the  Capability  Cfk  may  be  transformed  as  follows: 

7l  9  V  1  9  A  (  1  9  V  1  9  A 


ITRL  _CMl) 


1 


m 


MD 


1  ^ 

—  >  tx 

.  Q 

t= 1 

1  ^ 

—  >  tx 

o  ^ 


MW) 


J 


Ml) 


f  1  9  A4 

**  MiXi) 


v9w 


1 

;=1 


1  y  V  1  9 

o  X  txMW)  7tX  ^ 


0>m 


y 


v9m 


v9« 


m 


i_VlIv  ' 

2J  02-,tXMDU) 

”  /=i 


#(/)  MV- 


1  9 

q  X  ty Mi) 

”  r=i 


#(2) 
AC 


v  9  (=1 


y 


1  9  j  » 

7 tX^(0(«»)  7 


v9m 


(5) 


Thus,  based  on  the  computation  of  the  SRL  with  these  decision  variables,  MO_SRL  belongs  to  the  class 
of  integer  valued  non-linear  problems.  For  a  system  with  n  technologies  containing  m  ( m<(n-l)n/2 ) 
distinct  integrations,  and  assuming  all  technologies  and  integrations  are  at  their  lowest  levels,  there  are 
9"+m  possible  solutions  to  Model  MO_SRL.  To  solve  the  model,  this  research  adopted  an  algorithm  that's 
termed  as  Multi-Objective  Probabilistic  Solution  Discovery  Algorithm  (MO-PSDA). 

4.  Illustrative  Example  of  Implementation  of  Results2 

Previously,  an  MFMC  system  was  examined  in  Tan,  Sauser,  and  Ramirez-Marquez  (2011b)  and  Sauser, 
Ramirez-Marquez,  Magnaye,  et  al.,  (2008).  This  research  used  that  system  to  demonstrate  the 
application  of  the  proposed  MO_SRL  model  and  compare  the  results  with  previous  models.  Exhibit  4 
represents  an  end-to-end  integration  of  command  and  control  capabilities  with  a  variety  of  unmanned 
vehicles  and  intelligence,  surveillance,  and  reconnaissance  sensor  packages.  These  elements  are  capable 
of  autonomous  operations  and  include  both  commercial  off-the-shelf  equipment  and  cutting  edge 
developments  networked  seamlessly  together  to  enhance  mission  effectiveness  and  efficiency.  As 
configured,  there  are  20  technologies  and  21  integrations  in  the  system,  within  which  there  are  three 
functions  and  nine  capabilities  to  be  delivered  after  the  deployment  of  the  system.  Each  box  represents 
a  technology,  and  the  number  in  the  shaded  square  next  to  the  box  is  the  TRL  value  of  the 
corresponding  technology.  The  shaded  circle  contains  the  IRL  value  of  the  corresponding  integration.  As 
aforementioned,  the  potential  level  for  each  TRL  and  IRL  is  bounded  by  its  current  value  (as  shown  in 
Exhibit  4)  and  the  highest  possible  value  of  9. 


2  This  section  is  detailed  in  Tan,  W.,  Sauser,  B.,  Ramirez-Marquez,  J.,  &  Magnaye,  R.  (2012).  Multi-objective 
optimization  in  multifunction  multicapability  systems  development  planning.  IEEE  Transactions  on  Systems,  Man, 
and  Cybernetics:  Systems,  (available  online  in  lEEEXplore  Early  View). 
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Exhibit  4:  System  Diagram 

The  intended  notional  functions  and  capabilities  of  this  system  are  shown  in  Exhibit  5,  where  only  the 
required  technologies  and  their  corresponding  integrations  are  considered  in  evaluating  the  maturity  of 
each  of  the  corresponding  capabilities  of  a  function.  As  a  summary,  there  are  basically  three  different 
functions  under  which  there  are  three,  five,  and  one  capabilities,  respectively.  These  three  functions  are 
Mine-Detection,  Mine-Neutralization,  and  Gun-Combat.  Within  each  function,  there  are  several 
capability  alternatives  to  ensure  the  success  of  the  corresponding  mission.  For  example,  for  the  Mine- 
Detection,  one  of  the  capabilities  is  to  pull  the  sensor  by  using  a  submarine  which  has  longer  endurance 
but  slower  speed,  and  another  capability  is  by  using  a  helicopter,  which  is  faster,  but  can  only  stay  out 
for  a  limited  amount  of  time.  Using  the  enhanced  SRL  definition,  estimates  of  the  actual  SRLs  at  different 
levels  can  be  obtained  before  the  system  is  actually  deployed. 

Applying  the  MFMC  SRL  Flierarchy  to  this  system,  the  capability  and  function  SRLs  are  obtained  as 
shown  on  the  top  of  each  capability  thread  in  Exhibit  5  (refer  to  Tan,  Sauser,  &  Ramirez-Marquez,  2011b, 
for  details  of  calculating  SRL).  The  function  SRLs  are  shaded  with  their  capability  SRLs.  With  the  given 
notional  numbers  for  TRLs  and  IRLs,  all  of  these  SRLs  fall  in  a  range  from  0.58  to  0.68  that  corresponds  to 
a  system  stage  in  the  Engineering  &  Manufacturing  Development  phase  per  the  SRL  correlation  to  a 
typical  system  development  lifecycle  (see  Exhibit  6).  According  to  Sauser,  Ramirez-Marquez,  Magnaye, 
et  al.  (2008),  the  main  assignments  during  this  phase  are  to  develop  system  capability,  reduce 
integration  and  manufacturing  risk,  ensure  operational  supportability,  reduce  logistics  footprint, 
implement  human  systems  integration,  design  for  production,  ensure  affordability  and  protection  of 
critical  program  information,  and  demonstrate  system  integration,  interoperability,  safety,  and  utility. 
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Exhibit  6.  Correlation  of  SRL  to  a  System  Development  Lifecycle 


Cost  (in  thousand  dollars) 

Level 

Level 

Level 

Level 

Level 

Level 

Level 

Level 
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Exhibit  7.  Cost  for  TRL  and  IRL  Upgrade 

As  mentioned  previously,  the  second  type  of  optimization  is  to  minimize  the  resource  consumption,  and 
the  resource  can  be  anything  that  is  consumed  to  advance  the  maturity  of  a  system  (e.g.,  dollars,  labor 
hours,  machinery,  etc.).  For  illustration  purposes,  this  research  considers  dollars  as  resource 
consumption  for  upgrading  TRLs  and  IRLs  as  shown  in  Exhibit  7.  For  example,  it  requires  $446,000  to 
move  Technology  1  from  its  current  level  of  8  to  the  next  level  of  9.  This  research  focuses  on  the 
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incremental  development  efforts  needed  to  advance  the  system  from  its  current  reference  point.  It  does 
not  consider  the  budget  invested  in  the  past  for  bringing  the  TRL  or  IRL  to  its  current  level.  As  stated  in 
Ramirez-Marquez  and  Sauser  (2009),  it  is  the  obligation  of  the  program  manager  to  obtain  these 
estimates  of  resource  consumption  as  accurately  as  possible.  It  must  be  noted,  however,  that  there  is 
ongoing  research  in  the  development  of  a  System  Maturity  Model  to  enhance  cost  modeling  capabilities 
for  better  understanding  of  the  true  lifecycle  cost  (Volkert,  Jackson,  &  Whitfield,  2010).  Nonetheless,  the 
notional  estimated  cost  equals  to  $51,153,000  to  completely  mature  the  whole  system. 

Since  MO_SRL  emphasizes  the  maturity  assessment  at  the  function  and  capability  levels,  in  order  to 
demonstrate  the  proposed  MO_SRL  model,  two  scenarios  are  assumed  and  examined-one  for  function 
and  another  for  capability  SRLs  of  interest.  These  are  described  separately  in  the  following  section. 

4.1.  Function  SRL  Maximization  and  Cost  Minimization 

In  this  scenario,  the  objectives  are  to  maximize  the  SRL  of  Functions  1  and  2  and  to  simultaneously 
minimize  the  cost  of  upgrading  the  SRLs.  Therefore,  the  MO_SRL  model  can  be  specified  as  follows: 

Model  MO_SRL_Function 

Max  {SRL_F1  (TRL,  IRL),  SRL_F2  (TRL,  IRL)} 

Min  (cost  (TRL,  IRL)} 

For  this  scenario,  the  MO-PSDA  algorithm  was  applied  10  times  with  different  initial  appearance 
probabilities  for  pttand  piJjt,  and  within  each  setting  of  initial  appearance  probabilities,  10  cycles  of  1000 
solutions  were  generated  and  analyzed.  Therefore,  approximately  105 solutions  were  compared  in  order 
to  search  for  the  Pareto  optimal  solutions.  Compared  with  the  possible  total  920+22 feasible  outcomes  in 
the  solution  space  of  this  system,  the  portion  that  was  searched  counts  less  than  0.1%  in  an  affordable 
time  period.  Out  of  these  105  solutions,  this  experiment  was  able  to  locate  644  Pareto  optimal  solutions, 
which  collectively  consist  of  the  Pareto  optimal  front  as  depicted  in  Exhibit  8.  Each  dot  in  the  figure 
represents  a  Pareto  optimal  solution  that  corresponds  to  a  development  strategy  for  the  further 
advancement  of  Functions  1  and  2  with  the  minimal  budget.  These  solutions  range  from  resource 
consumption  of  $0— that  is,  investing  no  money— to  the  amount  of  $41,789,000,  which  can  improve  the 
SRLs  of  Functions  1  and  2  to  a  fully  mature  level  of  1. 
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Pareto  Optimal  Solutions  for  Budget  and  Functions  1  &  2 


Exhibit  8.  Pareto  Front  for  Budget  and  Functions  1  &  2 

Since  SRL  can  be  used  to  determine  the  development  status  of  a  system  (e.g.,  determining  the 
development  phase  the  system  is  currently  in),  this  experiment  is  further  interested  in  discovering 
strategies  that  can  improve  either  Function  1  or  Function  2  or  both  functions  to  more  mature 
development  phases  with  minimum  cost.  Thus,  with  the  consideration  of  the  correlation  of  SRL  to 
system  development  lifecycle,  as  shown  in  Exhibit  6,  this  experiment  was  able  to  locate  seven  Pareto 
optimal  strategies  that  will  advance  Function  1,  Function  2,  or  both  to  pass  development  milestones 
with  minimum  cost.  These  seven  strategies  are  depicted  in  Exhibit  9,  and  the  cost  for  executing  each 
development  strategy  is  also  attached  to  each  solution.  For  example,  in  order  to  mature  both  Function  1 
and  Function  2  to  pass  the  milestone  from  the  Engineering  and  Manufacturing  Development  phase  (SRL 
between  0.5  and  0.8,  see  Exhibit  6)  to  the  Production  and  Deployment  phase  (SRL  between  0.8  and  0.9, 
see  Exhibit  6),  the  minimum  cost  will  be  $10,410,000. 
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Pareto  Optimal  Solutions 


0.6  0.7  0.8  0.9  1 

SRL  of  Function  1 


Exhibit  9.  Pareto  Optimal  Solutions  with  Development  Milestones  (Cost  is  in  1000s) 

Since  no  solution  dominates  another  with  respect  to  all  the  objectives  of  the  model,  any  solution  is  not 
necessarily  better  than  another  one,  so  managers  have  flexibility  in  selecting  the  solution  that  best 
meets  their  goal  in  terms  of  advancing  system  maturity.  For  example,  the  manager  may  start  with  a 
preconceived  rough  budget,  look  through  the  solutions  with  the  needed  money  that  falls  within  his  or 
her  range,  and  then  balance  among  the  objectives  to  select  a  satisfactory  development  strategy  to 
mature  the  system. 

To  explain  the  result,  we  take  one  solution,  which  is  the  upper  right  solution  as  shown  in  Exhibit  9,  as  an 
example  to  drill  down  the  development  strategy  to  the  levels  of  technologies  and  integrations.  This 
strategy  will  bring  Function  1  and  Function  2  from  their  current  phase  (i.e.,  Engineering  and 
Manufacturing  Development)  to  the  Operations  and  Support  phase  with  a  minimum  resource 
consumption  of  $15,525,000. 

As  illustrated  in  Exhibit  10,  the  grayed  cells  show  that  the  corresponding  technologies  and  integrations 
need  to  be  advanced  from  their  current  status  to  an  expected  readiness  level  as  specified  in  the  table. 
For  example,  Technology  1  should  be  advanced  from  TRL  8  to  9  under  this  development  strategy.  The 
clear  area  of  Exhibit  10  indicates  that  these  technologies  and  integrations  will  stay  on  their  current 
status,  that  is,  no  further  budget  should  be  allocated  on  them.  Exhibit  11  shows  the  capability  and 
function  SRLs  under  this  strategy  as  well  as  the  cost  to  achieve  them.  The  grayed  cells  are  the  objectives 
of  this  scenario,  which  are  the  cost  and  SRLs  for  Functions  1  and  2.  This  table  shows  that  if  this  strategy 
is  adopted,  with  a  budget  of  $15,525,000,  SRLs  for  Functions  1  and  2  will  be  upgraded  to  above  0.9. 
Exhibit  11  also  shows  the  resulting  SRLs  for  other  capabilities  and  functions. 


16 


Technology 

Technology 

D 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Upgrade  to  TRL 

9 

9 

9 

9 

9 

9 

8 

6 

9 

Current  TRL 

8 

6 

5 

7 

6 

6 

8 

6 

8 

Technology 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

Upgrade  to  TRL 

6 

8 

7 

6 

9 

8 

9 

8 

8 

8 

Current  TRL 

6 

8 

7 

6 

8 

7 

6 

8 

7 

8 

Integration 

Integration 

■g 

2,3 

EM 

Ka 

4,19 

5,6 

6,8 

6,9 

8,10 

Upgrade  to  IRL 

9 

9 

9 

8 

6 

5 

7 

D! 

9 

8 

6 

Current  IRL 

D 

8 

8 

7 

6 

5 

7 

6 

8 

7 

6 

Integration 

9,11 

16,18 

17,11 

17,12 

17,13 

17,14 

17,15 

17,4 

18,20 

19,17 

20,4 

Upgrade  to  IRL 

5 

8 

5 

6 

8 

7 

8 

6 

7 

8 

6 

Current  IRL 

5 

8 

5 

6 

8 

7 

8 

5 

7 

7 

6 

Exhibit  10.  Development  Strategy  of  One  Example  Solution 
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Exhibit  11.  SRLs  and  Cost 


4.2.  Capability  SRL  Maximization  and  Cost  Minimization 

In  this  scenario,  the  objectives  are  to  maximize  the  SRLs  of  Capability  1.1  and  Capability  1.2,  and  to 
minimize  the  cost  of  upgrading  SRLs.  Thus,  the  MO_SRL_Capability  model  can  be  written  as  follows: 

Model  MO_SRL_Capability 

Max  {SRL_C1.1  (TRL,  IRL),  SRL_C1.2  (TRL,  IRL)} 

Min  {cost  (TRL,  IRL)} 

Running  the  same  experiment  scheme  as  the  above  scenario,  this  experiment  located  337  solutions  in 
the  final  Pareto  optimal  set.  These  solutions  consist  of  the  Pareto  front  that  is  visualized  in  Exhibit  12, 
with  resource  consumption  ranging  from  $0— that  is,  requiring  no  improvement  action— to 
$23,402,000,  which  advances  the  SRLs  of  Capabilities  1.1  and  1.2  to  a  complete  mature  level  of  1.  Each 
solution  informs  a  Pareto  optimal  strategy  for  advancing  the  system's  maturity. 

With  the  consideration  of  the  correlation  of  SRL  to  system  development  lifecycle,  this  example  was  able 
to  locate  five  Pareto  optimal  strategies  that  will  advance  Capability  1.1,  Capability  1.2,  or  both  to  pass 
development  milestones  with  minimum  cost.  These  five  strategies  are  depicted  in  Exhibit  13,  and  the 
cost  for  executing  each  development  strategy  is  also  attached  to  each  solution.  For  example,  in  order  to 
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mature  both  Capability  1.1  and  Capability  1.2  to  pass  the  milestone  to  the  Operations  and  Support 
phase  (SRL  between  0.9  and  1.0,  see  Exhibit  6),  the  minimum  cost  will  be  $11,590,000. 


Pareto  Optimal  Solutions  for  Budget  and  Capabilities  1.1  &  1.2 


Exhibit  12.  Pareto  Optimal  Solutions  for  Cost  and  Capabilities  1.1  &  1.2 
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Exhibit  13.  Pareto  Optimal  Solutions  with  Development  Milestones 

One  solution  corresponds  to  the  development  strategy  of  advancing  both  capabilities  from  the 
Engineering  and  Manufacturing  Development  phase  to  the  Production  and  Deployment  phase  is  drilled 
down  as  shown  in  Exhibit  14,  in  that  the  grayed  cells  indicate  the  corresponding  technologies  and 
integrations  needed  to  be  advanced  from  their  current  readiness  levels  to  an  expected  readiness  level  as 
specified  in  the  table.  Meanwhile,  the  other  technologies  and  integrations  will  stay  on  their  current 
status.  Capability  and  function  SRLs  under  this  strategy,  as  well  as  the  cost  to  achieve  them,  are  listed  in 
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Exhibit  15,  where  the  grayed  cells  are  the  objectives  of  this  scenario-the  cost  and  SRLs  for  Capabilities 
1.1  and  1.2.  Exhibit  15  indicates  that  if  this  strategy  is  adopted,  with  a  budget  of  $6,294,000,  SRLs  for 
both  Capabilities  1.1  and  1.2  will  be  upgraded  to  above  0.8. 
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Exhibit  14.  Development  Strategy  of  One  Example  Solution 
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Exhibit  15.  SRLs  and  Cost 

5.  Managerial  Implications  and  Future  Research 

This  research  extends  the  research  on  how  best  to  allocate  scarce  resources  during  the  development  of 
complex  systems.  Currently  available  optimization  models  as  proposed  rely  on  the  use  of  the  System 
Readiness  Level  (SRL)  scale  as  a  measure  of  the  readiness  of  a  whole  system.  This  SRL  scale  has  been 
mapped  notionally  to  existing  system  development  lifecycles  such  as  the  ones  being  used  for  space  and 
defense  systems.  Preliminary  results  appear  to  be  encouraging.  However,  the  SRL  scale  was 
conceptualized  to  be  applicable  to  the  whole  systems,  which  do  not  take  into  account  system  functions 
and  capabilities  in  the  development  of  MFMC  systems.  With  this  in  mind,  an  enhanced  SRL  concept 
called  the  multi-function  multi-capability  SRL  (MFMC  SRL)  has  also  been  proposed.  It  measures  the 
maturity  of  the  system  including  the  maturity  of  the  functions  and  capabilities  themselves  (Tan,  Sauser, 
&  Ramirez-Marquez,  2011b).  Since  these  systems  are  designed  to  perform  multiple  functions  and 
capabilities,  the  development  process  will  have  to  contend  with  competing  desires  to  advance  the 
maturity  of  several  functions  or  capabilities,  if  not  all  of  them,  at  once.  This  is  what  the  research 
addresses. 
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Using  the  enhanced  MFMC  SRL  Hierarchy,  this  research  proposes  a  multi-objective  optimization  model, 
which  can  be  used  to  allocate  development  resources  such  that  the  objectives  of  the  development 
strategy  are  met.  The  model,  referred  to  as  MO_SRL,  is  used  to  evaluate  the  development  of  a  notional 
sample  case,  which  is  a  system  intended  to  perform  three  functions  and  nine  capabilities  relying  on  20 
technology  elements  with  21  integration  links  among  them.  Two  scenarios,  both  with  three  objectives, 
were  assumed  and  analyzed  to  tailor  the  model  for  function  and  capability  SRL  maximization, 
respectively.  With  the  use  of  the  Multi-objective  Probabilistic  Solution  Discovery  Algorithm  (MO-PSDA), 
an  evolutionary  algorithm,  this  research  was  able  to  find  a  number  of  Pareto  optimal  solutions  for  each 
scenario.  These  indicate  to  which  level  of  readiness  each  system  component  (technology  element  or 
integration  link)  must  be  advanced  to  achieve  the  development  objectives  (readiness  of  functions  and 
capabilities)  and  how  much  they  will  cost. 

One  aspect  to  be  emphasized  is  that  the  proposed  model  only  applies  to  coherent  systems  in  which 
every  technology  has  to  be  integrated  with  the  rest  of  the  system,  no  matter  if  there  are  multiple 
interfaces  or  just  one  link.  As  a  first  step  to  address  multi-objective  optimization  in  system  maturity 
assessment  and  advancement,  model  MO_SRL  comes  with  some  limitations.  Future  work  will  further 
the  current  research  by  addressing  the  following  issues. 

First,  it  should  be  pointed  out  that  while  the  case  example  in  this  report  did  not  consider  the  situation  in 
which  multiple  technologies  with  the  same  functionality,  but  different  readiness  levels,  are  competing 
for  their  adoption  by  the  system,  the  proposed  model  has  the  potential  to  be  expanded  for  decision¬ 
making  in  such  a  situation.  That  is,  engineering  managers  can  evaluate  the  impact  of  trading  off  one 
technology  for  another  on  system  readiness.  The  results  presented  in  this  report  will  serve  as  a  basis  for 
exploring  a  more  complicated  situation  of  decision-making  on  system  development. 

Second,  the  model  in  the  report  appears  as  an  open  model  without  constraints.  This  is,  however,  not  a 
necessary  format,  and  the  model  can  also  be  applied  for  MO  problems  with  partial  constraints.  Our 
future  research  will  address  similar  problems  with  MO  but  partial  constraints. 

The  MO-PSDA  has  been  adopted  to  solve  the  MO_SRL  model  because  of  its  capability  to  efficiently  find 
converged  optimal  solutions  for  MO  optimization  problems.  However,  the  initialization  of  the 
appearance  probability  influences  the  direction  in  the  search  for  optimal  solutions.  Although  this 
research  employed  a  means  to  combine  the  results  from  several  initial  appearance  probabilities,  further 
research  is  needed  to  investigate  and  explore  a  sound  approach  for  selecting  the  probability. 

The  MO_SRL  model  is  based  on  the  assumption  that  there  are  available  data  on  resource  consumption 
for  upgrading  each  TRL/IRL  level.  As  the  premise  for  running  the  proposed  model,  reliable  resource 
estimation  plays  a  vital  role  in  the  viability  of  the  solutions  generated  by  the  MO_SRL  model.  Although  in 
practice,  it  mainly  depends  on  the  system  managers  and  developers  to  provide  these  data,  effective 
models  developed  in  academia  can  help  these  individuals  to  achieve  reliable  estimation.  Therefore, 
future  research  can  devote  efforts  to  this  area  (e.g.,  developing  cost  estimation  models). 

As  previously  mentioned,  the  SRL  methodology  has  gained  acceptance  in  study  and  practice;  however,  it 
needs  to  establish  statistical  effectiveness  through  rigorous  testing  with  systems.  In  general,  the 
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effectiveness  of  the  SRL  methodology  as  a  risk  management  tool  for  system  development  can  be  greatly 
enhanced  by  a  thorough  verification  and  validation  with  a  wide  cross-section  of  technologies  and 
systems  across  relevant  domains  (e.g.,  strategic  national  defense,  aerospace,  software,  energy, 
transport,  environment,  and  economic  systems).  Future  work  is  needed  for  both  cross-sectional  and 
longitudinal  research  with  case  studies  for  this  purpose. 

6.  Knowledge  Transfer  to  Industry/Government 

The  SysDML  is  currently  developing  a  Java-based  tool  that  will  work  with  three  commercially  available 
systems  architecting  tools  (i.e.,  Enterprise  Architect,  Systems  Architect,  and  Rhapsody).  This  effort  is 
funded  by  the  University-Affiliated  Research  Center  in  Systems  Engineering  (SERC).  The  objective  of  this 
development  is  to  translate  into  practice  academic  research  results  from  the  SysDML  via  a  practitioner 
tool.  A  first  version  of  an  SRL  Calculator  has  been  completed  that  works  with  Enterprise  Architect  on 
Windows  XP,  Vista,  and  7.  This  tool  is  being  tested  with  Northrop  Grumman  and  the  U.S.  Army  ARDEC.  A 
prototype  of  an  SRL  Calculator  that  will  work  with  Rhapsody  was  completed  in  August  2011.  Also,  the 
SysDML  has  established  a  partnership  with  ViTech  Corporation,  developers  of  the  systems  engineering 
architectural  tool(  i.e.,  CORE)  to  integrate  SysDML  research  results  into  their  commercial  offering  of 
CORE.  Our  industry  and  government  partners  have  agreed  to  work  with  the  SysDML  and  ViTech  in 
testing  any  prototypes  developed  by  ViTech. 

The  success  of  our  previous  research  and  of  this  proposed  research  is  contingent  on  our  strong  active 
stakeholder  relationships.  These  relationships  are  designed  to  accumulate  research  data  and  help  us 
keep  an  application  focus  to  the  practitioner  needs  and  alignment  with  acquisition  activities.  In  addition, 
all  of  the  reductions  to  practice  of  our  research  have  been  through  pilot  programs  within  the 
representative  organizations.  We  have  always  encouraged  this,  as  each  organization  needs  to  become 
comfortable  with  the  methods  as  well  as  assure  they  are  viable  and  reliable  for  use  with  existing 
programs.  Exhibit  16  is  an  example  of  some  of  our  active  stakeholders  and  the  value  of  the  relationship. 
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Active  Stakeholder 

Research  Value 

Lockheed  Martin 

The  implementation  into  practice  of  the  SysDML  methods  has  allowed  the  refinement  of 
current  industry  methods  to  incorporate  research-derived  insights,  provide  expanded 
functionality,  and  demonstrate  the  utility  of  the  method  in  the  context  of  a  multiple  pilot 
projects.  Lockheed  Martin  has  developed  their  own  version  of  an  SRL  Calculator  that  will 
work  with  Rhapsody  and  has  been  able  to  actively  utilize  the  tool. 

Northrop  Grumman 

A  long-term  relationship  with  NGC  has  resulted  in  significant  reduction  to  practice  and  the 
recent  release  of  Littoral  Combat  Ship  Mission  Module  Program  System  Maturity 
Assessment  Guide,  which  details  the  application  of  research  from  the  SysDML.  They  have 
also  incorporated  work  from  this  research  into  their  program  management  tools. 

NAVSEA/PMS  420 

Working  closely  with  PMS  420  has  allowed  for  the  leveraging  of  the  SRL  body  of  knowledge 
to  explore  the  effects  of  technology  and  integration  maturity  on  systems  engineering  effort 
and  cost;  ability  to  expand  the  scope  and  applicability  of  the  SRL  to  address  System  of 
Systems  methods,  processes,  and  tools;  and  the  enhancement  of  current  SRL  methods  and 
tools  to  incorporate  research-derived  insights,  provide  expanded  functionality,  and 
demonstrate  the  utility  of  the  tools  in  the  context  of  a  pilot  project. 

U.S.  Army  Armament 

Research  Development  and 
Engineering  Center  (ARDEC) 

ARDEC  has  been  a  longstanding  collaborator  with  the  SysDML  and  are  currently  developing 
internal  procedures  to  implement  the  SRL  and  other  research  results  into  practice. 

ViTech  Corporation 

The  SysDML  and  ViTech  Corporation  have  entered  into  a  partnership  that  will  transition 
research  results  into  future  releases  of  ViTech's  systems  engineering  architecture  tool,  i.e. 
CORE.  We  are  also  advising  ViTech  on  how  the  results  should  be  interpreted  and  presented 
within  a  systems  architecting  tool. 

Exhibit  16.  Active  Stakeholders  and  Research  Value 
7.  Project  Accomplishments 

7.1.  Publications 

7. 1. 1.  Journal 

Tan,  W.,  Sauser,  B.,  Ramirez-Marquez,  J.,  &  Magnaye,  R.  (2012).  Multi-objective  optimization  in 
multifunction  multicapability  systems  development  planning.  IEEE  Transactions  on  Systems, 

Man,  and  Cybernetics,  Part  A:  Systems,  (accepted) 

7.1.2.  Conference  Proceedings 

Sarfaraz,  M.,  &  Sauser,  B.  (2012,  March).  Improving  system  maturity  assessment  using  system 

engineering  architectures.  Paper  presented  at  the  Conference  on  Systems  Engineering  Research, 
St.  Louis,  MO 

7.1.3.  Working  Papers 

Magnaye,  R.,  Sauser,  B.,  &  Pantanakul,  P.  (2011).  Earned  readiness  management  for  scheduling, 
monitoring  and  evaluating  the  development  of  systems,  (working  paper) 
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Sauser,  B.  J.,  Magnaye,  R.,  Tan,  W.,  Ramirez-Marquez,  J.,  &  Sauser,  B.  W.  (2012).  Optimization  of 
system  maturity  and  equivalent  system  mass  for  space  systems  engineering  management. 
International  Journal  of  Space  Science  and  Engineering,  (under  review) 

7.2.  Presentations 

Sauser,  B.,  &  Ramirez-Marquez,  J.  (2012,  May  16-17).  Multi-objective  optimization  of  system 
capability  satisficing  in  defense  acguisition.  Paper  presented  at  the  Ninth  Annual  Acquisition 
Research  Symposium,  Monterey,  CA. 

8.  Other  Related  Activities 

8.1.  WebPage 

From  the  birth  of  this  research,  we  have  believed  in  an  open  academic  model  of  sharing  our  research 
outcomes  in  the  broadest  sense  possible.  Thus,  we  have  sustained  our  website 
(http://www.SysDML.com)  for  the  distribution  of  our  research  results.  At  this  website,  you  will  find  a 
research  overview,  a  list  of  publications  and  research  projects,  tools,  and  a  "Who  We  Are"  section. 

8.2.  Student  Research  Supported/Supervised 

Our  funding  from  the  Naval  Postgraduate  School  has  afforded  us  the  privilege  to  support  one  PhD 
student  to  assist  in  the  execution  of  this  research.  It  has  also  allowed  us  the  ability  to  attract  graduate 
students  to  pursue  related  and  supportive  research.  These  students  are  as  follows: 

Ivonne  Donate.  PhD  Student,  (supported  by  Department  of  Defense)  "Evolutionary  Lifecycle 
Assessment  for  Disruptive  Technology  Integration" 

Ryan  Gove.  PhD  Student,  (supported  by  Lockheed  Martin)  "Model  Based  Systems  Engineering  for 
Effective  System  Maturity  Assessment" 

Samuel  Russell.  PhD  Student,  (supported  by  NASA)  "The  Thermodynamics  of  System  Development:  A 
Mechanistic  Approach  to  System  Maturity  Assessment" 

Matin  Sarfaraz.  PhD  Student,  (supported  by  the  Innovation  and  Entrepreneurship  Graduate 
Fellowship )  "Systems  Engineering  Artifact  Correlation  to  Systems  Maturity  Assessment" 

Joseph  Uzdzinski.  (supported  by  Lockheed  Martin)  "System  Maturity  Assessment  for  Dynamic 
Interoperability  in  Complex  Systems" 

Graduates 

Weiping  Tan.  PhD  Student,  (supported  by  NPS)  "Methodologies  for  Component  Importance  Analysis 
in  Multi-Function,  Multi-Capability  System  Developmental  Maturity  Assessment" 

8.3.  Student  Awards 

School  of  Systems  and  Enterprises  Best  Student  Paper  Award,  Weiping  Tan,  2011 

For  a  graduate  student  who  has  distinguished  themselves  in  the  development  of  scholarly  work 
through  the  publication  of  a  peer-reviewed  conference  or  journal  paper:,  "Analyzing  Component 
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Importance  in  System  Maturity  Assessment,"  IEEE  Transactions  on  Engineering  Management, 
58(2),  275-294. 
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